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Amendment to the Claims 

This listing of claims will replace all prior versions, and listings, of claims in the application: 
Listing of Claims: 

CLAIM 1 (currently amended): A media and gesture recognition method using a computer 
system, the method comprising: 

viewing and generating a digital representation of a printed media using an electronic 

visual sensor during a first interaction session; 
identifying the printed media using the digital representation of the printed media; 
retrieving information corresponding to the viewed printed media fi-om a computer 
system database; 

using the electronic visual sensor to view at least a first gesture of a user relative to at 

least a portion of the printed media; 
interpreting the first fing e r gesture as a first command; and 

based at least in part on the first gesture and the retrieved information, providing at least 
a portion of the retrieved information. 

CLAIM 2 (original): The method as defined in claim 1, wherein identif)dng the printed media 
further comprises recognizing visual features that correspond to scale-invariant features (SIFT). 

CLAIM 3 (original): The method as defined in claim 1, wherein the electronic visual sensor is 
moxmted on a robot, wherein the robot positions itself so as to adequately view the printed media. 

CLAIM 4 (original): The method as defined in claim 1, wherein the electronic visual sensor is 
automatically tilted to improve the viewing of the printed media. 

CLAIM 5 (original): The method as defined in claim 1, fijrther comprising performing gesture 
calibration. 
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CLAIM 6 (original): The method as defined in claim 1, further comprising performing color 
balancing calibration based at least in part on a viewed portion of a userhand, 

CLAIM 7 (original): The method as defined in claim 1, further comprising instructing the user 
to perform at least one gesture during a calibration operation. 

CLAIM 8 (original): The method as defined in claim 1, vv^herein the first gesture is a diagonal 
sw^eep of a fingertip across a page of the printed media. 

CLAIM 9 (original): The method as defined in claim 1, wherein the first gesture is a movement 
of a fingertip beneath at least a first word. 

CLAIM 10 (original): The method as defined in claim 1, wherein the first gesture is a finger 
tapping movement. 

CLAIM 1 1 (original): The method as defined in claim 1, wherein the portion of the retrieved 
information is a word from the printed media. 

CLAIM 12 (original): The method as defined in claim 1, wherein the portion of the retrieved 
information is a sentence from the printed media. 

CLAIM 13 (original): The method as defined in claim 1, wherein the portion of the retrieved 
information is a title of the printed media. 

CLAIM 14 (original): The method as defined in claim 1, wherein the portion of the retrieved 
information is a table contents corresponding to the printed media. 

CLAIM 15 (original): The method as defined in claim 1, wherein the portion of the retrieved 
information includes a definition retrieved from an electronic dictionary. 
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CLAIM 16 (original): The method as defined in claim 1, wherein the printed media is one of a 
book, a magazine, a musical score, and a map. 

CLAIM 17 (original): The method as defined in claim 1, fiirther comprising: 

detecting an exception condition caused by an inadequate view of the printed media; and 
providing the user with instructions on handling the printed media to correct the 
exception condition. 

CLAIM 18 (original): The method as defined in claim 1, fiarther comprising: 
determining that the printed media is inadequately viewed; and 
instructing the user to rotate the printed media. 

CLAIM 19 (original): The method as defined in claim 1, fiirther comprising: 
detecting a timeout condition; and 

based at least in part on detecting the timeout condition, informing the user that the first 
interaction session is ended. 

CLAIM 20 (original): The method as defined in claim 1, wherein the database includes a 
preference that controls user interaction with the printed media at least at a book-level and a 
page-level, and a mapping of regions of the printed media with corresponding actions. 

CLAIM 21 (original): The method as defined in claim 1, fiirther comprising detecting the first 
gesture by comparing at least a first image and a second image electronic received by the visual 
sensor. 

CLAIM 22 (original): The method as defined in claim 1, wherein the visual sensor includes at 
least one of CCD imager, a CMOS imager, and an infrared imager. 

CLAIM 23 (original): A vision-based method of processing user interaction with printed media, 
the method comprising: 
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receiving at a computer system a digital representation of a first image of a printed 

media, wherein the first image was obtained fi"om a first imaging device; 
based at least in part on the digital representation of the first image, retrieving 

corresponding information from a database; 
receiving a first digital representation of a first image of a user gesture relative to at least 

a portion of the printed media; 
interpreting the first digital representation of an image of a user gesture; and 
based at least in part on the interpretation of the user gesture and the retrieved database 

information, providing at least a portion of the retrieved information to the user. 

CLAIM 24 (original): The method as defined in claim 23, wherein interpreting the digital 
representation of an image of a user gesture further comprises: 

finding averages for corresponding blocks within the first digital representation of the 

first image of the user gesture; 
subtracting the averages from averages of a prior digital representation of an image to 

generate a difference matrix having difference blocks; 
discarding difference blocks having averages beneath a first predetermined threshold; 
and 

discarding difference blocks having averages above a second predetermined threshold. 

CLAIM 25 (original): The method as defined in claim 23, wherein the user gesture is used to 
select printed media text and wherein providing at least a portion of the retrieved information to 
the user includes reading aloud the selected text. 

CLAIM 26 (original): The method as defined in claim 23, wherein the user gesture is used to 
select a printed image in the printed media and wherein providing at least a portion of the 
retrieved information to the user includes displaying a video related to the printed image. 

CLAIM 27 (original): The method as defined in claim 23, wherein the user gesture is used to 
select a map location in the printed media, and wherein providing at least a portion of the 
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retrieved information to the user includes providing information related to geographical location 
correspond to the selected map location. 

CLAIM 28 (original): The method as defined in claim 23, wherein the user gesture is used to 
select a portion of a musical score in the printed media, and wherein providing at least a portion 
of the retrieved information to the user includes audibly playing the selected portion of the 
musical score. 

CLAIM 29 (original): The method as defined in claim 23, wherein the first imaging device is 
mounted on an autonomous mobile apparatus, the method further comprising automatically 
positioning the autonomous mobile apparatus based on at least one image of the printed media. 

CLAIM 30 (original): The method as defined in claim 23, further comprising performing 
lighting calibration. 

CLAIM 3 1 (original): The method as defined in claim 23, further comprising providing the user 
with one or more audible media interaction prompts. 

CLAIM 32 (original): The method as defined in claim 23, further comprising: 
providing the user with a first prompt; 

waiting a first amount of time for the user to respond to the first prompt; and 
performing a timeout process if the user does not respond within the first amount of 
time, 

CLAIM 33 (original): The method as defined in claim 23, further comprising: 
determining if the printed media is skewed; and 
providing the user with skew correction prompts. 

CLAIM 34 (original): The method as defined in claim 23, further comprising: 
determining if the printed media is moving; and 
providing the user with an instruction to stop moving the media. 
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CLAIM 35 (original): The method as defined in claim 23, further comprising: 

determining if at least a first page of the printed media is not within a first image fi^ame; 
and 

informing the user that the system cannot view the entire page. 

CLAIM 36 (original): A computer-based printed media interaction apparatus, the apparatus 
comprising: 

an image sensor, the image sensor configured to view printed media; 
a database including a mapping of regions of the printed media with corresponding 
actions; 

a gesture tracking module that tracks a user gesture position relative to the printed media 

based at least in part on images from the image sensor; and 
an interaction module that, based at least in part on the user gesture position and 

database information, provides at least a portion of the database information to the 

user. 

CLAIM 37 (original): The apparatus as defined in claim 36, further comprising a plurality of 
motorized wheels under computer control used to position the image sensor to view the printed 
media. 

CLAIM 38 (original): The apparatus as defined in claim 36, further comprising an exception 
module that informs the user when the printed media is not being adequately viewed by the 
image sensor. 

CLAIM 39 (original): The apparatus as defined in claim 36, further comprising an exception 
module that informs the user when the printed media is moved. 

CLAIM 40 (original): The apparatus as defined in claim 36, wherein the gesture tracking 
module determines a difference between at least two images and filters out difference values 
greater than a first amount and difference values less than a second amount. 
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CLAIM 41 (original): The apparatus as defined in claim 36, wherein the image sensor is a pan 
and scan camera. 

CLAIM 42 (original): The apparatus as defined in claim 36, wherein the gesture tracking 
module determines if the user is making at least one of a read a word gesture and a read a page 
gesture. 

CLAIM 43 (original): The apparatus as defined in claim 36, wherein the gesture tracking 
module determines if the gesture corresponds to a request for a word definition. 

CLAIM 44 (original): The apparatus as defined in claim 36, ftirther comprising a dictionary. 

CLAIM 45 (original): The apparatus as defined in claim 36, fiirther comprising a topic-specific 
dictionary. 

CLAIM 46 (original): The apparatus as defined in claim 36, fiirther comprising a network link 
to information corresponding to the printed media. 

CLAIM 47 (original): The apparatus as defined in claim 36, fiirther comprising a speaker that 
audibly provides the database information to the user. 

CLAIM 48 (original): The apparatus as defined in claim 36, fiirther comprising a display that 
visually provides the database information to the user. 

CLAIM 49 (original): The apparatus as defined in claim 36, wherein the printed media is one of 
a magazine, a musical score, and a book. 

CLAIM 50 (original): The apparatus as defined in claim 36, fiirther comprising a character 
recognition module that converts images of text into text. 
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CLAIM 51 (original): A media and gesture recognition apparatus, the apparatus comprising: 
an image sensor that views printed media; 

a recognition module that identifies the printed media based on image information fi-om 
the image sensor; 

a database that stores information that relates portions of the printed media with 

corresponding actions; 
a gesture tracking module that identifies user gestures relative to the printed media based 

at least in part on images from the image sensor; and 
an interaction module that, based at least in part on the user gesture and database 

information, provides at least a portion of the database information to the user. 

CLAIM 52 (original): The apparatus as defined in claim 51, wherein the apparatus is stationary. 

CLAIM 53 (original): The apparatus as defined in claim 51, wherein the apparatus includes 
computer controlled motors that move the apparatus to view the printed media. 

CLAIM 54 (original): The apparatus as defined in claim 51, fiirther comprising a print media 
support apparatus. 

CLAIM 55 (original): The apparatus as defined in claim 51, wherein the database includes text 
fi-om the printed media, the apparatus further comprising a speaker that audibly reads at least a 
portion of the text to the user. 

CLAIM 56 (original): The apparatus as defined in claim 51, fiirther comprising a character 
recognition module that converts images of text into text. 

CLAIM 57 (original): The apparatus as defined in claim 51, further comprising a dictionary. 
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